Evaluating Spiking Neural Network On Neuromorphic Platform For Human Activity Recognition

Energy efficiency and low latency are crucial requirements for designing wearable AI-empowered human activity recognition systems, due to the hard constraints of battery operations and closed-loop feedback. While neural network models have been extensively compressed to match the stringent edge requirements, spiking neural networks and event-based sensing are recently emerging as promising solutions to further improve performance due to their inherent energy efficiency and capacity to process spatiotemporal data in very low latency. This work aims to evaluate the effectiveness of spiking neural networks on neuromorphic processors in human activity recognition for wearable applications. The case of workout recognition with wrist-worn wearable motion sensors is used as a study. A multi-threshold delta modulation approach is utilized for encoding the input sensor data into spike trains to move the pipeline into the event-based approach. The spikes trains are then fed to a spiking neural network with direct-event training, and the trained model is deployed on the research neuromorphic platform from Intel, Loihi, to evaluate energy and latency efficiency. Test results show that the spike-based workouts recognition system can achieve a comparable accuracy (87.5\%) comparable to the popular milliwatt RISC-V bases multi-core processor GAP8 with a traditional neural network ( 88.1\%) while achieving two times better energy-delay product (0.66 \si{\micro\joule\second} vs. 1.32 \si{\micro\joule\second})

Fully Automatic Gym Exercises Recording: An IoT Solution

In recent years, working out in the gym has gotten increasingly more data-focused and many gym enthusiasts are recording their exercises to have a better overview of their historical gym activities and to make a better exercise plan for the future. As a side effect, this recording process has led to a lot of time spent painstakingly operating these apps by plugging in used types of equipment and repetitions. This project aims to automate this process using an Internet of Things (IoT) approach. Specifically, beacons with embedded ultra-low-power inertial measurement units (IMUs) are attached to the types of equipment to recognize the usage and transmit the information to gym-goers and managers. We have created a small ecosystem composed of beacons, a gateway, smartwatches, android/iPhone applications, a firebase cloud server, and a dashboard, all communicating over a mixture of Bluetooth and Wifi to distribute collected data from machines to users and gym managers in a compact and meaningful way. The system we have implemented is a working prototype of a bigger end goal and is supposed to initialize progress toward a smarter, more efficient, and still privacy-respect gym environment in the future. A small-scale real-life test shows 94.6\% accuracy in user gym session recording, which can reach up to 100\% easily with a more suitable assembling of the beacons. This promising result shows the potential of a fully automatic exercise recording system, which enables comprehensive monitoring and analysis of the exercise sessions and frees the user from manual recording. The estimated battery life of the beacon is 400 days with a 210 mAh coin battery. We also discussed the shortcoming of the current demonstration system and the future work for a reliable and ready-to-deploy automatic gym workout recording system

ERICA: Enabling real-time mistake detection and corrective feedback for free-weights exercises

National Research Foundation (NRF) Singapore under International Research Centres in Singapore Funding Initiativ

ColibriES: A Milliwatts RISC-V Based Embedded System Leveraging Neuromorphic and Neural Networks Hardware Accelerators for Low-Latency Closed-loop Control Applications

End-to-end event-based computation has the potential to push the envelope in latency and energy efficiency for edge AI applications. Unfortunately, event-based sensors (e.g., DVS cameras) and neuromorphic spike-based processors (e.g., Loihi) have been designed in a decoupled fashion, thereby missing major streamlining opportunities. This paper presents ColibriES, the first-ever neuromorphic hardware embedded system platform with dedicated event-sensor interfaces and full processing pipelines. ColibriES includes event and frame interfaces and data processing, aiming at efficient and long-life embedded systems in edge scenarios. ColibriES is based on the Kraken system-on-chip and contains a heterogeneous parallel ultra-low power (PULP) processor, frame-based and event-based camera interfaces, and two hardware accelerators for the computation of both event-based spiking neural networks and frame-based ternary convolutional neural networks. This paper explores and accurately evaluates the performance of event data processing on the example of gesture recognition on ColibriES, as the first step of full-system evaluation. In our experiments, we demonstrate a chip energy consumption of 7.7 \si{\milli\joule} and latency of 164.5 \si{\milli\second} of each inference with the DVS Gesture event data set as an example for closed-loop data processing, showcasing the potential of ColibriES for battery-powered applications such as wearable devices and UAVs that require low-latency closed-loop control

ColibriUAV: An Ultra-Fast, Energy-Efficient Neuromorphic Edge Processing UAV-Platform with Event-Based and Frame-Based Cameras

The interest in dynamic vision sensor (DVS)-powered unmanned aerial vehicles (UAV) is raising, especially due to the microsecond-level reaction time of the bio-inspired event sensor, which increases robustness and reduces latency of the perception tasks compared to a RGB camera. This work presents ColibriUAV, a UAV platform with both frame-based and event-based cameras interfaces for efficient perception and near-sensor processing. The proposed platform is designed around Kraken, a novel low-power RISC-V System on Chip with two hardware accelerators targeting spiking neural networks and deep ternary neural networks.Kraken is capable of efficiently processing both event data from a DVS camera and frame data from an RGB camera. A key feature of Kraken is its integrated, dedicated interface with a DVS camera. This paper benchmarks the end-to-end latency and power efficiency of the neuromorphic and event-based UAV subsystem, demonstrating state-of-the-art event data with a throughput of 7200 frames of events per second and a power consumption of 10.7 \si{\milli\watt}, which is over 6.6 times faster and a hundred times less power-consuming than the widely-used data reading approach through the USB interface. The overall sensing and processing power consumption is below 50 mW, achieving latency in the milliseconds range, making the platform suitable for low-latency autonomous nano-drones as well

Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO

Smart glasses are rapidly gaining advanced functionality thanks to cutting-edge computing technologies, accelerated hardware architectures, and tiny AI algorithms. Integrating AI into smart glasses featuring a small form factor and limited battery capacity is still challenging when targeting full-day usage for a satisfactory user experience. This paper illustrates the design and implementation of tiny machine-learning algorithms exploiting novel low-power processors to enable prolonged continuous operation in smart glasses. We explore the energy- and latency-efficient of smart glasses in the case of real-time object detection. To this goal, we designed a smart glasses prototype as a research platform featuring two microcontrollers, including a novel milliwatt-power RISC-V parallel processor with a hardware accelerator for visual AI, and a Bluetooth low-power module for communication. The smart glasses integrate power cycling mechanisms, including image and audio sensing interfaces. Furthermore, we developed a family of novel tiny deep-learning models based on YOLO with sub-million parameters customized for microcontroller-based inference dubbed TinyissimoYOLO v1.3, v5, and v8, aiming at benchmarking object detection with smart glasses for energy and latency. Evaluations on the prototype of the smart glasses demonstrate TinyissimoYOLO's 17ms inference latency and 1.59mJ energy consumption per inference while ensuring acceptable detection accuracy. Further evaluation reveals an end-to-end latency from image capturing to the algorithm's prediction of 56ms or equivalently 18 fps, with a total power consumption of 62.9mW, equivalent to a 9.3 hours of continuous run time on a 154mAh battery. These results outperform MCUNet (TinyNAS+TinyEngine), which runs a simpler task (image classification) at just 7.3 fps per second

Human Activity Recognition with Field Sensing Technique

The development of machine learning algorithms and novel sensing modalities has boosted the exploration of human activity recognition(HAR) in recent years. In this work, we explored field-based sensing solutions and different machine learning models for HAR tasks to address the shortcomings of existing HAR sensing solutions, like the weak robustness of RF-based solution, environment-dependency of the optic-based solution, etc., aiming to supply a competitive and alternative sensing approach for HAR tasks. Field, in physics, describes a region in which each point will be affected by force. Field sensing is potentially a low-cost, low-power, non-intrusive, privacy-respecting HAR solution that is ideal for long-term, wearable activity recording. By directly/indirectly monitoring the field strength or other field variation caused variables, some unsolved HAR problems could be addressed when other sensing solutions fail. An example is the social distance monitoring problem, where the most widely adopted approach is based on the Bluetooth signal strength measurement. However, the signal is so subtle that any object surrounding the signal emitter will cause signal attenuation. To guarantee the accuracy of social distance monitoring, we developed an induced magnetic field-based social distance monitoring system with an accuracy of a sub-ten centimetre. Moreover, the system is robust and resistant to environmental variations. Like Bluetooth, other RF-wave-based sensing modalities also face the multi-path effect caused by refraction. Thus their signal is unreliable for positioning applications where higher accuracy and robustness are needed. Besides the magnetic field, we also explored a natural static passive electric field, the field between the human body and surroundings, namely the human body capacitance(HBC). HBC is a physiological parameter describing the charge distribution difference between the body and the surroundings and is seldomly explored before. We developed a few wearable, low-cost, low power consumption hardware platforms, either based on an oscillating unit or discrete components composed sensing front end followed by a high resolution analog-to-digital module, to monitor the variation of the parameter regarding the body movement and environmental variations. Compared with the inertial sensors, the HBC could deliver full-body movement perceiving, meaning that the movement of the legs could be perceived by a wrist-worn HBC sensing unit, which is far beyond the sensing ability of an inertial sensing unit. To summarize, we introduced two competitive field sensing modalities for HAR tasks, the magnetic field sensing for position-related services and the passive electric field sensing for full-body action and environmental variation sensing. Both of which were still in an infant stage and not fully explored in the community. The advantages of the two field sensing modalities were demonstrated with a series of position-related and motion-related experiments

Human Activity Recognition with Field Sensing Technique

The development of machine learning algorithms and novel sensing modalities has boosted the exploration of human activity recognition(HAR) in recent years. In this work, we explored field-based sensing solutions and different machine learning models for HAR tasks to address the shortcomings of existing HAR sensing solutions, like the weak robustness of RF-based solution, environment-dependency of the optic-based solution, etc., aiming to supply a competitive and alternative sensing approach for HAR tasks. Field, in physics, describes a region in which each point will be affected by force. Field sensing is potentially a low-cost, low-power, non-intrusive, privacy-respecting HAR solution that is ideal for long-term, wearable activity recording. By directly/indirectly monitoring the field strength or other field variation caused variables, some unsolved HAR problems could be addressed when other sensing solutions fail. An example is the social distance monitoring problem, where the most widely adopted approach is based on the Bluetooth signal strength measurement. However, the signal is so subtle that any object surrounding the signal emitter will cause signal attenuation. To guarantee the accuracy of social distance monitoring, we developed an induced magnetic field-based social distance monitoring system with an accuracy of a sub-ten centimetre. Moreover, the system is robust and resistant to environmental variations. Like Bluetooth, other RF-wave-based sensing modalities also face the multi-path effect caused by refraction. Thus their signal is unreliable for positioning applications where higher accuracy and robustness are needed. Besides the magnetic field, we also explored a natural static passive electric field, the field between the human body and surroundings, namely the human body capacitance(HBC). HBC is a physiological parameter describing the charge distribution difference between the body and the surroundings and is seldomly explored before. We developed a few wearable, low-cost, low power consumption hardware platforms, either based on an oscillating unit or discrete components composed sensing front end followed by a high resolution analog-to-digital module, to monitor the variation of the parameter regarding the body movement and environmental variations. Compared with the inertial sensors, the HBC could deliver full-body movement perceiving, meaning that the movement of the legs could be perceived by a wrist-worn HBC sensing unit, which is far beyond the sensing ability of an inertial sensing unit. To summarize, we introduced two competitive field sensing modalities for HAR tasks, the magnetic field sensing for position-related services and the passive electric field sensing for full-body action and environmental variation sensing. Both of which were still in an infant stage and not fully explored in the community. The advantages of the two field sensing modalities were demonstrated with a series of position-related and motion-related experiments

Social Distance Monitor with a Wearable Magnetic Field Proximity Sensor

Social distancing and contact/exposure tracing are accepted to be critical strategies in the fight against the COVID-19 epidemic. They are both closely connected to the ability to reliably establish the degree of proximity between people in real-world environments. We proposed, implemented, and evaluated a wearable proximity sensing system based on an oscillating magnetic field that overcomes many of the weaknesses of the current state of the art Bluetooth based proximity detection. In this paper, we first described the underlying physical principle, proposed a protocol for the identification and coordination of the transmitter (which is compatible with the current smartphone-based exposure tracing protocols). Subsequently, the system architecture and implementation were described, finally an elaborate characterization and evaluation of the performance (both in systematic lab experiments and in real-world settings) were performed. Our work demonstrated that the proposed system is much more reliable than the widely-used Bluetooth-based approach, particularly when it comes to distinguishing between distances above and below the 2.0 m threshold due to the magnetic field’s physical properties

The State-of-the-Art Sensing Techniques in Human Activity Recognition: A Survey

Human activity recognition (HAR) has become an intensive research topic in the past decade because of the pervasive user scenarios and the overwhelming development of advanced algorithms and novel sensing approaches. Previous HAR-related sensing surveys were primarily focused on either a specific branch such as wearable sensing and video-based sensing or a full-stack presentation of both sensing and data processing techniques, resulting in weak focus on HAR-related sensing techniques. This work tries to present a thorough, in-depth survey on the state-of-the-art sensing modalities in HAR tasks to supply a solid understanding of the variant sensing principles for younger researchers of the community. First, we categorized the HAR-related sensing modalities into five classes: mechanical kinematic sensing, field-based sensing, wave-based sensing, physiological sensing, and hybrid/others. Specific sensing modalities are then presented in each category, and a thorough description of the sensing tricks and the latest related works were given. We also discussed the strengths and weaknesses of each modality across the categorization so that newcomers could have a better overview of the characteristics of each sensing modality for HAR tasks and choose the proper approaches for their specific application. Finally, we summarized the presented sensing techniques with a comparison concerning selected performance metrics and proposed a few outlooks on the future sensing techniques used for HAR tasks